Inverse-Reference Priors for Fisher Regularization of Bayesian Neural Networks

Recent studies have shown that the generalization ability of deep neural networks (DNNs) is closely related to Fisher information matrix (FIM) calculated during early training phase. Several methods been proposed regularize FIM for increased DNNs. However, they cannot be used directly Bayesian (BNNs) because variable parameters BNNs make it difficult calculate FIM. To address this problem, we achieve regularization by specifying a new suitable prior distribution called inverse-reference (IR) prior. FIM, IR derived as inverse reference imposes minimal knowledge on and maximizes trace We demonstrate can enhance large-scale data over previously priors while providing adequate uncertainty quantifications using various benchmark image datasets BNN structures.